Control apparatus, control method, image pickup apparatus, and storage medium

ABSTRACT

A control apparatus is configured to control a display unit configured to superimpose and display a captured image captured by an image sensor and display items indicating in-focus degrees of a plurality of focus detecting areas in the captured image. The control apparatus includes a memory that stores a set of instructions, and at least one processor that executes the set of instructions to change a display state of the display items according to a state of a manual focus operation by a user.

BACKGROUND Technical Field

One of the aspects of the embodiments relates to a control apparatus, adisplay control method, and a storage medium.

Description of Related Art

A display apparatus that displays a focus state of an object (anin-focus state, a front focus state, a rear focus state, or a defocusdegree) based on a focus evaluation value of the object has recentlybeen proposed as a focus assisting function in a manual focus (MF)operation.

Japanese Patent No. 6537325 discloses a display control apparatusconfigured to display an area having a predetermined size forcalculating a focus evaluation value around a user specified position asa center on an imaging screen (image), and a guide index that indicatesa focus state based on the focus evaluation value in this area. JapanesePatent Laid-Open No. 2021-117469 discloses a display control apparatusthat displays an in-focus area (in-focus frame) on the imaging screenduring autofocus (AF). Japanese Patent Laid-Open No. 2010-97167discloses an AF apparatus that displays an AF frame by setting an objectlocated at the best focus position to an AF target at the end of afocusing operation, and starts automatically tracking the object.

The display control apparatus disclosed in Japanese Patent No. 6537325cannot recognize a focus state of the entire imaging screen or an areaother than a predetermined area around the user specified position asthe center. Therefore, in a case where the object is moving or there area plurality of objects, focusing by the MF operation is difficult. Thedisplay control apparatus disclosed in Japanese Patent Laid-Open No.2021-117469 has difficulty in focusing because the distance anddirection to the in-focus position are unknown, and if many focusingframes are displayed on the object, it becomes difficult to confirm thefocus state of the object. The AF apparatus disclosed in Japanese PatentLaid-Open No. 2010-97167 has difficulty in providing the object with thebest focus state.

SUMMARY

One of the aspects of the embodiments provides a control apparatus thatcan achieve both improved focusing operability during an MF operationand secured the visibility during focus confirmation of an object.

A control apparatus according to one aspect of the disclosure isconfigured to control a display unit. The display unit is configured tosuperimpose and display a captured image captured by an image sensor anddisplay items indicating in-focus degrees of a plurality of focusdetecting areas in the captured image. The control apparatus includes amemory that stores a set of instructions, and at least one processorthat executes the set of instructions to change a display state of thedisplay items according to a state of a manual focus operation by auser. A control method corresponding to the above control apparatus alsoconstitutes another aspect of the disclosure. A storage medium storing aprogram that causes a computer to execute the above control method alsoconstitutes another aspect of the disclosure. An image pickup apparatushaving the above control apparatus also constitutes another aspect ofthe disclosure.

Further features of the disclosure will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging system according to thisembodiment.

FIG. 2 is a pixel array diagram on an image sensor according to thisembodiment.

FIG. 3 is a flowchart illustrating focus assisting frame displayprocessing in this embodiment.

FIGS. 4A, 4B, and 4C explain phase-difference detecting frame settingprocessing according to this embodiment.

FIG. 5 is a flowchart illustrating phase-difference detecting processingaccording to this embodiment.

FIG. 6 is a flowchart illustrating MF assisting frame setting processingaccording to this embodiment.

FIG. 7 is a flowchart illustrating MF assisting frame display processingaccording to this embodiment.

FIGS. 8A to 8F explain the display of MF assisting frames and MFassisting subframes in this embodiment.

FIGS. 9A and 9B explain a display range of the MF assisting frames inthis embodiment.

FIG. 10 explains the delay time until the assisting frame is displayedin this embodiment.

FIG. 11 is a flowchart illustrating AF assisting frame settingprocessing according to this embodiment.

FIGS. 12A and 12B are flowcharts illustrating AF assisting frame displayprocessing according to this embodiment.

FIG. 13 explains an AF frame displaying menu according to thisembodiment.

FIGS. 14A to 14H explain the AF assisting frame and the MF assistingframe in this embodiment.

FIGS. 15A to 15E explain connected display of the AF assisting frameaccording to this embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following, the term “unit” may refer to a software context, ahardware context, or a combination of software and hardware contexts. Inthe software context, the term “unit” refers to a functionality, anapplication, a software module, a function, a routine, a set ofinstructions, or a program that can be executed by a programmableprocessor such as a microprocessor, a central processing unit (CPU), ora specially designed programmable device or controller. A memorycontains instructions or programs that, when executed by the CPU, causethe CPU to perform operations corresponding to units or functions. Inthe hardware context, the term “unit” refers to a hardware element, acircuit, an assembly, a physical structure, a system, a module, or asubsystem. Depending on the specific embodiment, the term “unit” mayinclude mechanical, optical, or electrical components, or anycombination of them. The term “unit” may include active (e.g.,transistors) or passive (e.g., capacitor) components. The term “unit”may include semiconductor devices having a substrate and other layers ofmaterials having various concentrations of conductivity. It may includea CPU or a programmable processor that can execute a program stored in amemory to perform specified functions. The term “unit” may include logicelements (e.g., AND, OR) implemented by transistor circuits or any otherswitching circuits. In the combination of software and hardwarecontexts, the term “unit” or “circuit” refers to any combination of thesoftware and hardware contexts as described above. In addition, the term“element,” “assembly,” “component,” or “device” may also refer to“circuit” with or without integration with packaging materials.

Referring now to the accompanying drawings, a detailed description willbe given of embodiments according to the disclosure.

Referring now to FIG. 1 , a description will be given of an imagingsystem according to this embodiment. FIG. 1 is a block diagram of theimaging system 10. The imaging system 10 has a focus assistingdisplaying function as described below. As illustrated in FIG. 1 , theimaging system 10 includes a camera body (image pickup apparatus) 111and a lens unit (lens apparatus) 100 attachable to and detachable fromthe camera body 111. A lens control unit 109 that controls the overalloperation of the lens unit 100 and a camera control unit 119 (controlapparatus) that controls the overall operation of the imaging system 10can communicate data with each other.

In this embodiment, the camera control unit 119 is a control unit havinga processor such as a CPU or MPU (processor) and a storage unit such asa memory. The camera control unit 119 may also have a calculationcircuit (processor), and may execute some of calculation functionsperformed by the processor using the calculation circuit. In thisembodiment, the lens unit 100 is an interchangeable lens attachable toand detachable from the camera body 111. However, this embodiment is notlimited to this example and is applicable to an image pickup apparatusin which the lens unit and the camera body are integrated.

A description will now be given of the configuration of the lens unit100. The lens unit 100 includes an imaging optical system that includesa first fixed lens 101, a zoom lens (magnification varying lens) 102, adiaphragm (aperture stop) 103, a second fixed lens 104, and a focus lens105. An image (optical image) of the object is formed by the imagingoptical system. The zoom lens 102 is a lens movable in a direction alongthe optical axis OA (optical axis direction) to perform a magnificationvarying (zooming) operation, and is driven by a zoom lens driving unit106 for zooming. The focus lens 105 serves to correct movement of thefocal plane caused by zooming and to provide a focusing function, and isdriven by a focus lens driving unit 108 for focusing. The diaphragm 103is driven by an aperture driving unit 107 to control a light amountincident on an image sensor 112, which will be described below.

The zoom lens driving unit 106, the aperture driving unit 107, and thefocus lens driving unit 108 are controlled by the lens control unit 109,which controls the aperture diameter in the diaphragm 103 and thepositions of the zoom lens 102 and the focus lens 105. In a case wherethe user operates a focus ring, a zoom ring, or the like (notillustrated) provided in the lens operation unit 110 to perform thefocusing or zooming operation, the lens control unit 109 performscontrol according to the user operation. The lens control unit 109controls the aperture driving unit 107, the zoom lens driving unit 106,and the focus lens driving unit 108 according to the control command andcontrol information received from the camera control unit 119, andtransmits lens information to the camera control unit 119.

A description will now be given of the configuration of the camera body111 having the focus assisting function according to this embodiment. Inthe camera body 111, the image sensor 112 includes a CCD sensor or aCMOS sensor, and a light beam that has passed through the imagingoptical system in the lens unit 100 forms an image on a light receivingplane (imaging plane) of the image sensor 112. Then, the formed objectimage is photoelectrically converted into charges according to anincident light amount by the photodiodes (photoelectric conversionunits) of the image sensor 112 and accumulated. The charges accumulatedin each photodiode are sequentially read out from the image sensor 112as a voltage signal corresponding to the charge based on the drivingpulse given from a timing generator (TG) 117 in accordance with thecommand from the camera control unit 119. The detailed configuration ofthe image sensor 112 will be described below, but the image sensor 112in this embodiment can output a pair of focusing signals for focusdetection by the imaging-plane phase-difference method in addition tonormal imaging signals.

The imaging signal and focus signal read out from the image sensor 112are input to a Correlated Double Sampling (CDS)/Auto Gain Control (AGC)circuit 113 to perform correlated double sampling for removing resetnoises, gain control, and signal digitization. The CDS/AGC circuit 113outputs the processed imaging signal to a camera signal processing unit114 and the focus signal to a focus signal processing unit 118.

The camera signal processing unit 114 performs various image processingfor an imaging signal output from the CDS/AGC circuit 113 and generatesa video signal. A display unit 115 is a display apparatus such as LCD ororganic EL, and displays an image based on the video signal output fromthe camera signal processing unit 114. The display unit 115 superimposesand displays a focus assisting frame (display item indicating thein-focus degree) according to this embodiment on the screen (image).That is, the display unit 115 superimposes and displays the capturedimage captured by the image sensor 112 and the display item indicatingthe in-focus degree of the plurality of focus detection areas in thecaptured image. Details of the focus assisting frame will be describedbelow. In this embodiment, the display unit 115 is included in thecamera body 111, but the disclosure is not limited to this embodiment,and the display unit may be a display apparatus such as a televisionmonitor connected as an external device to the camera body 111. In arecording mode for recording the imaging signal, the imaging signal issent from the camera signal processing unit 114 to a recorder (REC) 116and recorded on a recording medium such as an optical disc, asemiconductor memory, or a magnetic tape.

The focus signal processing unit 118 performs correlation calculationbased on the pair of focus signals output from the CDS/AGC circuit 113and detects the focus state. In this embodiment, the focus signalprocessing unit 118 calculates a correlation amount, a defocus amount,and reliability information (two-image matching degree, two-imagesteepness degree, contrast information, saturation information, flawinformation, etc.). The focus signal processing unit 118 then outputs acalculated defocus amount and reliability information to the cameracontrol unit 119. Based on the defocus amount or the reliabilityinformation acquired from the focus signal processing unit 118, thecamera control unit 119 notifies the focus signal processing unit 118 ofchanges in settings for calculating them.

The camera control unit 119 control each component in the camera body111 by exchanging information with it. The camera control unit 119controls processing within the camera body 111, powering on and off,changes in imaging settings and display settings, and recording of dataaccording to input from the camera operation unit 120 operated by theuser. The camera control unit 119 executes various functions accordingto user operations such as AF/MF control switching and recorded videoconfirmation. The camera control unit 119 communicates information withthe lens control unit 109 in the lens unit 100, transmits a controlcommand and control information for the imaging optical system, andacquires information in the lens unit 100, as described above. As willbe described below, the camera control unit 119 changes the displaystate of display items in accordance with the MF operation of the user.

In this embodiment, the display control apparatus may include at leastthe display unit 115 and the camera control unit 119.

Referring now to FIG. 2 , a description will be given of the pixel arrayof the image sensor 112 in this embodiment. FIG. 2 is a pixel arraydiagram of the image sensor 112 and illustrates a range of 4 columns×4rows of imaging pixels (8 columns×4 rows for the array of focusdetecting pixels) as the pixel array of the two-dimensional CMOS sensorfor the image sensor 112.

This embodiment assumes that a pixel unit 200 consists of 2 columns×2rows of pixels and covered with color filters in a Bayer array. In thepixel unit 200, a pixel unit 200R having R (red) spectral sensitivity islocated at the upper left position, pixel units 200G having G (green)spectral sensitivity are located at the upper right and lower leftpositions, and a pixel unit 200B having B (blue) spectral sensitivity islocated at the lower right position.

Since the image sensor 112 performs focus detection using theimaging-plane phase-difference method, each pixel in the image sensor112 holds a plurality of photodiodes (photoelectric converters) for asingle microlens (not illustrated). In this embodiment, each pixelincludes two photodiodes 201 and 202 arranged in 2 columns×1 row. Theimage sensor 112 can acquire an imaging signal and a focus signal byarranging a large number of pixel units 200 each consisting of 4columns×4 rows of pixels (8 columns×4 rows of photodiodes) illustratedin FIG. 2 .

In each pixel having such a configuration, a light beam is separated bya microlens and imaged on photodiodes 201 and 202. A signal (A+B signal)obtained by adding the signals from the two photodiodes 201 and 202 isused as an imaging signal, and two signals (A and B image signals) readout of the individual photodiodes 201 and 202 are used as focusingsignals. The imaging signal and the focusing signal may be read outseparately, but this embodiment may perform as follows based on theprocessing load. That is, this example may read out the imaging signal(A+B signal) and one of the focusing signals (such as A signal) of thephotodiodes 201 and 202, calculate a difference, and acquire the otherfocusing signal (such as B signal).

In this embodiment, each pixel has two photodiodes 201 and 202 for asingle microlens, but the number of photodiodes is not limited to two,and may be more. A plurality of pixels having different openingpositions of the light receiving portions relative to the microlens maybe provided. In other words, any configuration may be used as long astwo phase-difference detecting signals such as an A image signal and a Bimage signal, which can provide phase difference detection, are obtainedas a result. This embodiment is not limited to the configuration inwhich every pixel has a plurality of photodiodes as illustrated in FIG.2 . The configuration illustrated in FIG. 2 in which focus detectingpixels are discretely provided may be employed in normal pixels on theimage sensor 112.

Referring now to FIGS. 3 to 10 , a description will be given of displayprocessing of the focus assisting frame (display item indicating thein-focus degree) executed by the camera control unit 119 according tothis embodiment. FIG. 3 is a flowchart illustrating the overall sequenceof focus assisting frame display processing executed by the cameracontrol unit 119. The processing of steps S301 to S308 described belowis periodically executed based on the operation cycle of the camera body111.

First, in step S301, the camera control unit 119 acquires various camerasettings such as imaging settings, display settings, AF/MF modesettings, etc. that have been changed according to inputs from thecamera operation unit 120 operated by the user. In this embodiment, thecamera operation unit 120 acquires, as camera settings, settingsrelating to the display of a focus assisting frame or a focus assistingsubframe, which will be described below, for display during MF. Forexample, camera settings can be changed such as a setting of whether ornot to always display the focus assisting frame or focus assistingsubframe, a setting of whether or not to display the focus assistingframe or focus assisting subframe only during user operation, or adisplay method (frame color, shape, and size, frame display range).After step S302, the camera control unit 119 performs processingaccording to the acquired camera setting. The camera control unit 119can set three patterns of the AF frame display menu in FIG. 13 for thedisplay during AF. Details of this will be described below withreference to FIG. 13 .

Next, in step S302, the camera control unit 119 performsphase-difference detecting frame setting processing. Thephase-difference detecting frame setting processing is settingprocessing of a detecting frame for acquiring a signal that is used forfocus detection in the imaging-plane phase-difference method. Details ofthe setting position of the phase-difference detecting frame will bedescribed below with reference to FIGS. 4A, 4B, and 4C. The descriptionof this embodiment assumes that a plurality of phase-differencedetecting frames are internally set by dividing a predeterminedphase-difference detecting range into grid-like areas, but thedisclosure is not limited to this example. The phase-differencedetecting range may set a size range according to the camera settingacquired in step S301 around a position as a center specified by theuser through a touch operation or the like.

Next, in step S303, the camera control unit 119 performsphase-difference detecting processing. In the phase-difference detectingprocessing, the camera control unit 119 performs focus detection usingthe imaging-plane phase-difference method for the set phase-differencedetecting frames, and calculates a detection result, such as a shiftamount (defocus amount) to an in-focus position and a shift direction(defocus direction) based on the focus detection result. Details of thephase-difference detecting processing will be described below withreference to FIG. 5 .

Next, in step S304, the camera control unit 119 performs settingprocessing of a frame for assisting focusing in the MF operation (MFassisting frame, display item indicating the in-focus degree) based onthe detection result of the phase-difference detecting processing instep S303. Next, in step S305, the camera control unit 119 performsprocessing to superimpose and display the MF assisting frame set in stepS304 on a captured image displayed on the display unit 115. Details ofsteps S304 and S305 will be described below with reference to FIGS. 6and 7 . Focusing on an object as an imaging target can become easier byvisually displaying the MF assisting frame or the MF assisting subframeon the display unit 115 such as a monitor based on the acquiredphase-difference detecting result, and by explicitly indicating anin-focus area of the camera body 111 to the user.

Next, in step S306, the camera control unit 119 determines whether ornot the mode is the AF mode according to the state of the focus switch(not illustrated) in the lens operation unit 110 of the lens unit 100 orthe camera operation unit 120 of the camera body 111. In a case wherethe camera control unit 119 determines that the mode is the AF mode, theflow proceeds to step S307. In a case where the camera control unit 119determines that the mode is not the AF mode (the mode is the MF mode),this flow ends.

In step S307, the camera control unit 119 performs setting processing ofa frame (AF assisting frame) for assisting smooth transition from the MFoperation to the AF operation based on the detection result of thephase-difference detecting processing in step S303. Next, in step S308,the camera control unit 119 performs processing to superimpose anddisplay the AF assisting frame set in step S307 on the display unit 115.

Details of steps S307 and S308 will be described with reference to FIGS.11 to 15E. This embodiment visually displays the AF assisting frame, theAF assisting subframe, or the like set based on the acquiredphase-difference detecting result on the display unit 115 such as amonitor, and explicitly indicates to the user a focusing area in the AFcontrol performed by the camera control unit 119. Thereby, in a casewhere the AF control takes over the MF operation, this configurationenables which object the AF control will start focusing on to be easilyrecognized.

Referring now to FIGS. 4A, 4B, and 4C, a description will be given ofthe phase-difference detecting frame setting processing in step S302 ofFIG. 3 . FIGS. 4A, 4B, and 4C explain the phase-difference detectingframe setting processing. FIG. 4A illustrates a phase-differencedetecting area 400 disposed in the pixel areas on the image sensor 112.The phase-difference detecting area 400 is an area in which the camerabody 111 performs focus detection. The description of this embodimentassumes that the phase-difference detecting area 400 is not explicitlydisplayed on the screen, but the frame may be displayed on the screen sothat the user can check it on a monitor or the like. While FIG. 4Aillustrates that the target is limited to the center portion of thepixel area on the image sensor 112, the entire pixel area of the imagesensor 112 may be targeted.

Phase-difference detecting frames 401 illustrated in FIG. 4B are set sothat detecting frames divided into N×M (11×11 in this embodiment) arearranged adjacent to each other inside the phase-difference detectingarea 400. However, this embodiment is not limited to this example.Neighboring detecting frames may be spaced by a certain distance, or maybe set so that the detecting frames partially overlap each other. Forexample, without changing the size (height and width) of thephase-difference detecting frame 401, neighboring detecting frames mayoverlap each other by ½ of the size of one frame in the left, right, up,and down directions. In this case, similarly to phase-differencedetecting frames 402 illustrated in FIG. 4C, the number of divisionsinside the phase-difference detecting area 400 can be substantially22×22.

Referring now to FIG. 5 , a description will be given of thephase-difference detecting processing in step S303 of FIG. 3 . FIG. 5 isa flowchart illustrating the phase-difference detecting processing.First, in step S501, the camera control unit 119 acquires a pair ofimage signals for each phase-difference detecting frame 401 placed inthe phase-difference detecting area 400 set in step S302. Next, in stepS502, the camera control unit 119 calculates a correlation amountbetween the pair of image signals acquired in step S501. Next, in stepS503, the camera control unit 119 calculates a correlation change amountbased on the correlation amount calculated in step S502. Next, in stepS504, the camera control unit 119 calculates a focus shift amount basedon the correlation change amount calculated in step S503. Next, in stepS505, the camera control unit 119 calculates reliability indicating howreliable the focus shift amount calculated in step S504 is. As describedabove, the reliability is a value calculated based on the two-imagematching degree or the two-image steepness degree of the image signals.

Next, in step S506, the camera control unit 119 converts the focus shiftamount calculated in step S504 into a defocus amount. Next, in stepS507, the camera control unit 119 determines whether or not theprocessing of steps S501 to S506 has been completed for all the 121phase-difference detecting frames set in step S302. In a case where theprocessing for all the phase-difference detecting frames has not beencompleted, the flow returns to step S501, and steps S501 to S506 arerepeated. On the other hand, in a case where the processing for all thephase-difference detecting frames is completed, this flow ends.

Referring now to FIGS. 6 to 10 , a detailed description will be given ofthe processing for displaying the MF assisting frame. FIG. 6 is aflowchart illustrating the MF assisting frame setting processing in stepS304 of FIG. 3 .

First, in step S601, the camera control unit 119 acquires detectioninformation on each phase-difference detecting frame. Here, thedetection information on each phase-difference detecting frame includesa focus state determined based on the reliability, defocus amount, anddefocus direction calculated for each phase-difference detecting framedescribed with reference to FIG. 5 . The focus state includes at leastthree states: an “in-focus state” indicating that the object is infocus, and a “front focus state” and a “rear focus state,” which areslightly out of focus, and are different in shifting direction from thein-focus position. For example, the camera control unit 119 determinesthe focus state based on whether the calculated reliability is high, andthe calculated defocus amount is smaller than an in-focus determinationthreshold (which is a defocus amount obtained by multiplying a defocusamount corresponding to a depth of field by a predetermined ratio). In acase where the defocus amount is smaller than the in-focus determinationthreshold, the camera control unit 119 determines the focus state to bethe “in-focus state.” This embodiment sets the predetermined ratio to 2,and the defocus amount that is twice as large as the depth of field tothe in-focus determination threshold. In a case where the defocus amountlarger than the in-focus determination threshold appears in the infinitydirection, the state in which the focus is on the close side (frontside) of the in-focus position is determined to be the “front focusstate.” In a case where the defocus amount larger than the in-focusdetermination threshold appears in the close direction, it is determinedthat the focus is on the infinity side (rear side) of the in-focusposition, and that the state is the “rear focus state.” In addition tothe three states of the “in-focus state,” the “front focus state,” andthe “rear focus state,” a “bokeh state” indicating that the image isblurred may be determined in a case where the calculated reliability islow or in a case where the defocus amount exceeds a predeterminedthreshold (such as a defocus value equivalent to five times as large asthe depth of field) defocus amount). This embodiment determines thefocus state to be the “front focus state” or the “rear focus state” in acase where the calculated defocus amount is larger than the in-focusdetermination threshold, but the focus state is not determined to be the“bokeh state.”

The depth of field that serves as a reference for the in-focusdetermination threshold is determined according to the state of theimaging optical system such as the zoom lens 102, the focus lens 105, orthe diaphragm 103. In general, a range of the in-focus determination iswider as a focal length becomes shorter, an imaging distance (distancebetween the object and the camera) becomes longer, and an F-number(aperture value) becomes larger. This embodiment changes the in-focusdetermination threshold for the defocus amount according to the depth offield. For example, in a case where the depth of field is larger than apredetermined value, the in-focus determination threshold for thedefocus amount is set to a value smaller than that where the depth offield is small. This is because if the in-focus determination thresholdis fixed, most of the phase-difference detecting frames are determinedto be the “focus state,” the “front focus state,” and the “rear focusstate” under wide-angle and narrowed-down conditions that increase thedepth of field. Changing the in-focus determination threshold canprevent the MF assisting frame or the MF assisting subframe described instep S603 and subsequent steps from being displayed on an area otherthan the object aimed by the user, such as the entire screen, and thevisibility from deteriorating.

The determining ranges for the “in-focus state,” the “front focusstate,” and the “rear focus state” may be changed according to thecamera setting obtained in step S301 in FIG. 3 as well as the depth offield. For example, the in-focus determination threshold may be selectedfrom among “wide,” “narrow,” “linked to the recorded size” in the camerasetting menu, and the predetermined ratio may be changed according tothis setting. This embodiment acquires, as the detection information,frame number information indicating how many frames there are for eachfocus state together with the focus state.

Referring now to FIGS. 8A to 8F, a description will be given of the MFassisting frames and the MF assisting subframes as display items. FIGS.8A to 8F explain the display of the MF assisting frames and the MFassisting subframes. FIGS. 8A to 8F illustrate the display state of theMF assisting frames and the MF assisting subframes in an attempt tofocus on a back object 802 while a front object 801 is in focus usingthe MF operation toward the infinity direction. FIGS. 8A and 8Eillustrate a state in which no MF operation is performed. FIGS. 8B, 8C,8D, and 8F illustrate a state in which the MF operation is beingperformed. FIG. 8B illustrates a state just after the MF operation isstarted. FIG. 8C illustrates a state in which an area between the frontobject 801 and the back object 802 is in focus while the MF operation isbeing performed. FIG. 8D illustrates a state in which the back object802 is in focus but the MF operation is still continued.

In FIGS. 8A to 8F, each rectangular frame 803 indicates an MF assistingframe, and each dotted-line frame 804 indicates an MF assistingsubframe. The MF assisting frame indicates the in-focus position on thescreen, and the MF assisting subframe indicates the almost (marginal)in-focus state (in the example of FIGS. 8A to 8F, since the MF operationis being performed in the infinity direction, the focus state is thefront focus state) so as to visually indicate their positions to theuser. A series of flows in FIGS. 8A to 8F is one of the characteristicsaccording to this embodiment.

In a case where the MF operation is not being performed, the displaydensity of the MF assisting frames is reduced and the visibility issecured in order to enable the focus state of the object to be easilyconfirmed by thinning out the MF assisting frames of the neighboringframes among the phase-difference detecting frames, by reducing the sizeof one frame, and by discretely displaying the neighboring frames, forexample, by displaying the neighboring frames with gaps. That is, thevisibility is secured by reducing the display ratio of the MF assistingframes per unit area. During the MF operation for focusing, the displaydensity of the MF assisting frames is increased by displaying allneighboring MF assisting frames and by increasing the size of one frameto close the gap. That is, the display ratio occupied by the MFassisting frame per unit area is increased.

Thus, the camera control unit 119 changes the display state of the MFassisting frame as a display item according to the MF operation of theuser. For example, the camera control unit 119 may set the display ratioto a first ratio in a case where the user is not performing the MFoperation, and may set a second ratio higher than the first ratio in acase where the user is performing the MF operation. The camera controlunit 119 may set the display number of the MF assisting frames to afirst display number in a case where the user is not performing the MFoperation, and may set the display number of the MF assisting frames toa second display number larger than the first display number in a casewhere the user is performing the MF operation. The camera control unit119 may set the display gap of the MF assisting frames to a first gap ina case where the user is not performing the MF operation, and may setthe display gap of the MF assisting frames to a second gap narrower thanthe first gap in a case where the user is performing the MF operation.Thereby, a focus assisting function can be realized that enables theuser to visually recognize changes in the in-focus position associatedwith focus changes, and perceive a remaining amount of the MF operationto make the targeted object in focus.

This embodiment indicates the MF assisting frame as a display item bythe rectangular frame 803 and the MF assisting subframe as a displayitem by the dotted-line frame 804, but is not limited to this example.For example, as in MF assisting subframes 805 in FIG. 8F, thisembodiment may change the attribute such as a frame color (displaycolor), transparency, and thickness between the MF assisting frame andthe MF assisting subframe, and a frame shape, such as a key shape, forone of the MF assisting frame and the MF assisting subframe. Thisembodiment may make different the MF assisting frame and the MFassisting subframe according to the camera setting acquired in step S301of FIG. 3 . This embodiment may enable a frame color (first color orsecond color) and a shape (first shape or second shape) to be selected:For example, the MF assisting frame may be set to a green rectangularframe and the MF assisting frame may be set to a yellow rectangularframe.

Now back to FIG. 6 , a description will be given of the flow after stepS602. In step S602, the camera control unit 119 determines whether ornot there is a focusing operation using a focus ring (not illustrated)on the lens operation unit 110, a focusing operation button (notillustrated) on the camera operation unit 120, or the like. In a casewhere there is a focusing operation, the flow proceeds to step S603. Ina case where there is no focusing operation, the flow proceeds to stepS604.

In step S603, the camera control unit 119 turns on a flag (MF assistingframe display flag) indicating whether or not to display the MFassisting frames for all of the phase-difference detecting frames in the“in-focus state” acquired in step S601. In step S604, the camera controlunit 119 turns on the MF assisting frame display flag for part of thephase-difference detecting frames in the “in-focus state” acquired instep S601. This embodiment thins out and displays neighboring MFassisting frames of the phase-difference detecting frames, but is notlimited to this example. As long as the visibility is secured during thefocus confirmation, another method of lowering the display density ofthe MF assisting frames (reducing the ratio of the MF assisting framesper unit area) may be used. For example, this embodiment can use amethod of thinning out and discretely displaying the MF assisting framesor a method of reducing the size of each frame and displaying the frameswith a gap.

The MF assisting frame and the MF assisting subframe may not match thesize of the phase-difference detecting frame. For example, in a casewhere the MF operation is not being performed in a large number ofdivisions, such as the phase-difference detecting frames 402 in FIG. 4C,the visibility may be secured by displaying four frames (a plurality ofdisplay items) that are vertically and horizontally adjacent to oneanother. Moreover, in a case where the MF operation is being performed,a plurality of frames may be displayed without being combined.

In step S605, the camera control unit 119 determines whether or not thenumber of frames determined to be “in focus” (the number of in-focusframes) among the detection information acquired in step S601 is smallerthan a predetermined number Nth. In a case where it is determined thatthe number of frames is smaller than the predetermined number Nth, theflow proceeds to step S606. In a case where it is determined that thenumber of frames is larger than the predetermined number Nth, the flowproceeds to step S609. The reason why whether or not to perform theprocessing of steps S606 to S608 is changed according to the number offocusing frames will be described below.

In step S606, the camera control unit 119 determines whether or not thefocusing operation is performed in the infinity direction. In a casewhere it is determined that the focusing operation is performed in theinfinity direction, the flow proceeds to step S607. In a case where itis determined that the focusing operation is performed in the closedirection, the flow proceeds to step S608.

In step S607, the camera control unit 119 turns on a flag (MF assistingsubframe display flag) indicating whether or not to display the MFassisting subframe in the phase-difference detecting frame determined tobe in the “front focus state.” In step S608, the camera control unit 119turns on the MF assisting subframe display flag of the phase-differencedetecting frame determined to be in the “rear focus state.” That is, thecamera control unit 119 sets the range of the in-focus degree as thedisplay state of the display item to a first range in a case where theuser is not performing the MF operation, and sets the range of thein-focus degree to a second range wider than the first range in a casewhere the user is performing the MF operation.

This embodiment displays the MF assisting subframe in either the “frontfocus state” or the “rear focus state” depending on the focusingoperation direction. In addition to a position in an “in-focus state,” aposition in an “almost in-focus state” may be recognized. If the MFassisting subframe is displayed on the bokeh frames, the screen becomesso crowded that focusing becomes difficult. However, this embodiment isnot limited to this example. This embodiment may always display the MFassisting subframes in the “front focus state” and “rear focus state”regardless of the focusing operation method, and may change the framecolor and frame shape so that the MF assisting subframes in the “frontfocus state” and “back focus state” can be distinguished.

The reason why steps S606 to S608 are not performed in a case where thenumber of in-focus frames is larger than the predetermined number Nth instep S605 is as follows. That is, in a case where the display density ofthe MF assisting frames is high, or in a case where there are manyin-focus frames on the screen, it becomes easier to visually view achange in an in-focus position along with a focus change, and the usercan perceive a remaining amount of the MF operation to make the targetedobject in focus. Thus, even if the MF assisting subframe is notdisplayed, focusing becomes easier to some extent. In addition, in acase where there are many in-focus frames displayed on the screen, ifthe MF assisting subframes are also displayed, the screen may becrowded, and the visibility may deteriorate. In this embodiment, thecamera control unit 119 determines whether or not to change the range ofthe in-focus degree according to the number of displayed MF assistingframes.

However, this embodiment is not limited to this example, and may setwhether or not to display the MF assisting subframe by the menu settingor the like, and always display the MF assisting subframe regardless ofthe number of focusing frames. For example, in a case where the camerasetting acquired in step S301 of FIG. 3 is a setting that does notalways display the MF assisting subframes, step S605 is alwaysestablished by setting the predetermined number Nth to a sum of themaximum number of phase-difference detecting frame 401 and 1 and thusthe MF assisting subframe is displayed. Conversely, a setting that doesnot display the MF assisting subframes can be set by setting thepredetermined number Nth to 0.

Referring now to FIGS. 9A and 9B, a description will be given of thedisplay range of the MF assisting frame or MF assisting subframe. FIGS.9A and 9B explain the display range of the MF assisting frame. FIG. 9Ais a conceptual diagram illustrating the target angle of view (grayrectangular frame in the figure) after the zoom operation and thepan/tilt operation are combined with the current angle of view. FIG. 9Bis a conceptual diagram illustrating the setting range of the MFassisting frame while the zoom operation and the pan/tilt operation arebeing performed toward the target angle of view. Similarly to FIG. 8D,FIGS. 9A and 9B, schematically illustrate a state in which the MFoperation has been performed in the infinity direction from a frontobject 901, and a back object 902 is in focus, but the MF operation isstill being continued. FIG. 9B illustrates that the display range of theMF assisting frame 903 is limited to a display range (dotted-line frame)904 instead of the entire screen in a case where the zoom operation andthe pan/tilt operation are also combined. During zooming to thetelephoto side (that is, during enlargement), an object in theperipheral portion of the screen becomes out of the angle of view, sothe display range 904 of the MF assisting frame may be limited to thecentral portion of the screen. Even in the pan/tilt operation, theobject in the direction opposite to the pan/tilt direction becomes outof the angle of view, so the display range 904 of the MF assisting framemay be shifted in the pan/tilt direction.

Now back to FIG. 6 , the flow after step S609 will be described. In stepS609, the camera control unit 119 determines whether there is a zoomoperation to the telephoto side, using the zoom ring (not illustrated)on the lens operation unit 110, the zoom operation button (notillustrated) on the camera operation unit 120, and the like. In a casewhere there is the zoom operation to the telephoto side, the flowproceeds to step S610. In a case where there is no zoom operation orthere is a zoom operation to the wide-angle side, the flow proceeds tostep S611. In step S610, the camera control unit 119 restricts thedisplay range of the MF assisting frame by setting it to a small sizebecause the object in the peripheral portion of the screen becomes outof the angle of view in a case where the zoom operation is performed tothe telephoto side as described above. That is, the camera control unit119 changes the display range of the MF assisting frames according tothe zoom operation of the user.

Next, in step S611, the camera control unit 119 determines whether ornot there is a pan/tilt operation (panning operation or tiltingoperation) of the camera body 111. This determination is made based onan output from a gyro sensor (not illustrated) mounted on the lens unit100 or the camera body 111, vector information detected from the videosignal from the camera signal processing unit 114, and the like. In acase where there is the pan/tilt operation, the flow proceeds to stepS612. In a case where there is no pan/tilt operation, this flow isterminated. In step S612, since the object in the direction opposite tothe pan/tilt direction becomes out of the angle of view as describedabove, the camera control unit 119 shifts the display range 904 of theMF assisting frame in the pan/tilt direction and sets it. The displayrange 904 of the MF assisting frame may be set so as to expand the rangein the pan/tilt direction instead of shifting in the pan/tilt direction.That is, the camera control unit 119 changes the display range of the MFassisting frame according to the panning operation or tilting operationof the user.

The shifting degree of the object from the angle of view changesdepending on the speed of the zoom operation or the pan/tilt operation.Therefore, the setting of the display range 904 of the MF assistingframe may be changed according to the zoom speed and the pan/tilt speed.For example, in a case where the zoom speed is low or in a case wherethe pan/tilt speed is low, the display range 904 of the MF assistingframe is not particularly limited. In a case where the zoom speed ishigh, the display range 904 of the MF assisting frame may be reduced andlimited. In a case where the pan/tilt speed is high, the display rangeof the MF assisting frame may be shifted or expanded in the pan/tiltdirection.

The display ranges of the MF assisting frame and MF assisting subframeassociated with the zoom operation and pan/tilt operation are notlimited to the above methods. The MF assisting frame and MF assistingsubframe may be always displayed on the entire screen regardless ofzooming or panning/tilting operations by making settable the displayrange of the MF assisting frame through the menu setting, etc.

Referring now to FIG. 7 , a description will be given of the MFassisting frame display processing in step S308 of FIG. 3 . FIG. 7 is aflowchart illustrating MF assisting frame display processing.

First, in step S701, the camera control unit 119 determines whether eachphase-difference detecting frame 401 is within the display range(display range 904 illustrated in FIG. 9B) of the MF assisting frame(and the MF assisting subframe) set in steps S610 and S612 of FIG. 6 .In a case where each phase-difference detecting frame 401 is within thedisplay range of the MF assisting frame, the flow proceeds to step S702.In a case where each phase-difference detecting frame 401 is outside thedisplay range of the MF assisting frame, the flow proceeds to step S706.

In step S702, the camera control unit 119 determines whether or not theMF assisting frame display flag set in steps S603 and S604 of FIG. 6 isturned on. In a case where the MF assisting frame display flag is turnedon, the flow proceeds to step S704. In a case where the MF assistingframe display flag is turned off, the flow proceeds to step S703.

In step S703, the camera control unit 119 determines whether the MFassisting subframe display flag set in steps S607 and S608 of FIG. 6 isturned on. In a case where the MF assisting subframe display flag isturned on, the flow proceeds to step S705. In a case where the MFassisting subframe display flag is turned off, the flow proceeds to stepS706.

In step S704, the camera control unit 119 superimposes the MF assistingframes on the screen. In step S705, the camera control unit 119superimposes the MF assisting subframes on the screen. In step S706, thecamera control unit 119 hides (does not display) the MF assisting framesand the MF assisting subframes.

Next, in step S707, the camera control unit 119 determines whether ornot the processing of steps S701 to S706 has been completed to all 121phase-difference detecting frames set in step S302. In a case where theprocessing to all phase-difference detecting frames has not beencompleted, the flow returns to step S701, and the processing of stepsS701 to S706 are repeated. In a case where the processing to all thephase-difference detecting frames is completed, this flow ends.

A description has been given hitherto of the way of facilitatingfocusing during the MF operation and securing the visibility during thefocus confirmation by changing the displays of the MF assisting framesand the MF assisting subframes according to the MF operation. However,in practice, it takes a long time to detect a phase difference betweenimaging signals and to determine the focus state. In displaying the MFassisting frames and the MF assisting subframes corresponding to thedetermined focus state on the display unit 115 such as an externalmonitor (not illustrated), there is a delay time from when the cameracontrol unit 119 sends the frame information to the display unit 115 towhen the notification is displayed. Hence, focusing on an objecttargeted by the user may become difficult without exceeding the focusposition by simply displaying the focus assisting frame according to thephase-difference detecting result.

FIG. 10 explains the delay time until the assisting frame is displayed.In FIG. 10 , a horizontal axis indicates a focus lens position p, and avertical axis indicates time. FIG. 10 illustrates a relationship betweena focus lens position and elapsed time in a case where the object 802 inFIGS. 8A to 8F is to be focused by the MF operation, and a relationshipbetween the range of the “in-focus state” (in-focus range) and the rangeof the “front focus state” (front focus range). The focus lens positionp indicates the position of the focus lens in a case where the object802 is in focus.

This embodiment displays the MF assisting frames and the MF assistingsubframes by determining the focus state based on the calculated defocusamount and defocus direction, but is not limited to this example. Here,for better understanding, the displays of the MF assisting frames andthe MF assisting subframes may be set based on the relationship betweenthe focus state and the time change of the remaining driving amount ofthe focus lens to the in-focus position, which is calculated based onthe defocus amount and the defocus direction. Therefore, the remainingdriving amount of the focus lens up to the in-focus position will beused for the description. For simplified description, this embodimentassumes that the driving amount of the focus lens by the MF operation isa constant speed.

A focus lens position p1 indicates a focus lens position at time t1 in acase where the defocus amount is calculated in the phase-differencedetecting processing in step S303 of FIG. 3 . A remaining focus drivingamount up to the in-focus position at time t1 is p−p1. A focus lensposition p2 indicates a focus lens position at time t2 in a case wherethe focus state is determined in the MF assisting frame settingprocessing of step S304 in FIG. 3 and the MF assisting frame displayflag is set based on this determination result. A period from the timet1 when the defocus amount is calculated to the time t2 when the focusstate is determined corresponds to calculation time ti1 for the focusstate determination. Therefore, the focus lens moves by p2−p1 during thecalculation time ti1. The remaining focus driving amount up to thein-focus position at time t2 is p−p2.

A focus lens position p3 indicates a focus lens position at time t3 whenthe assisting frame is displayed on the screen of the display unit 115.There is delay time ti2 from the time t2 when the camera control unit119 determines the focus state to the time t3 when the assisting frameis actually displayed on the display unit 115 due to the communicationdelay time for notifying information on the display of the assistingframe to the display unit 115, and the response time of the display unit115 itself. Therefore, the focus lens moves by p3−p2 during the delaytime ti2. The remaining focus driving amount up to the in-focus positionat time t3 is p−p3, but the assisting frame displayed at time t3 isdisplayed based on the focus state at the time t1 when the defocusamount is calculated. When the user visually perceives the assistingframe display, the focus driving amount p3−p1 during the calculationtime ti1 and the delay time ti2 is not considered.

The user stops the MF operation after viewing the assisting framesdisplayed on the display unit 115, but it is expected that it takes timeti3 from time t3 when the assisting frames are displayed to time t4 whenthe user reacts to the display and actually stops the focusingoperation. Even in a case where the user stops the focusing operation,there is actually an idle running time ti4 from when the focus lensdecelerates to when it stops at time t5. Therefore, in order to stop thefocus lens at the in-focus position p or within the in-focus range, theuser is previously informed of the “in-focus state.”

This embodiment displays the remaining focus driving amount up to thein-focus position for displaying the assisting frames and the assistingsubframes, that is, the defocus amount by subtracting a moving amountduring a period of a sum of the calculation time ti1, the delay timeti2, the time ti3, and the idle running time ti4. The calculation timeti1 and the idle running time ti4 are determined according to theconfiguration of the imaging system 10 that includes the lens unit 100and the camera body 111. Hence, a predetermined value is set based onthe previously measured time or the like. The delay time ti2 isdetermined according to the communication system between the cameracontrol unit 119 and the display unit 115 and the responsiveness of thedisplay unit 115. Therefore, the predetermined value is set based on thepreviously measured time. Alternatively, in a case where the camera body111 is connected to an external monitor or the like, the delay timevaries depending on the connected monitor, so a menu setting may beprovided so that the user can specify the delay time. Since the time ti3from when the user views the display to when the user reacts to itdiffers from person to person, the user may be able to specify it.

As described above, the camera control unit 119 can change the displayof the display item according to the focus control amount by the MFoperation by acquiring a first offset amount for the focus detectionresult, and by subtracting the first offset amount from the focusdetection result. The camera control unit 119 may change the display ofthe display item by acquiring a second offset amount according to adisplay delay amount of the display item on the display unit 115, and bysubtracting the second offset amount from the focus detection result.The camera control unit 119 may change the display of the display itemby acquiring a third offset amount set by the user and by subtractingthe third offset amount from the focus detection result.

As described above, this embodiment secures the visibility by loweringthe display density of the MF assisting frames in order to make iteasier to check the focus state of the object in a case where no MFoperation is being performed. During the MF operation for focusing, thedisplay density of the MF assisting frames is increased. Thereby, theuser can visually recognize changes in the in-focus position along withfocus changes, and perceive a remaining amount of the MF operation tomake the targeted object in focus. This embodiment can provide a focusassisting display apparatus that can achieve improved focusingoperability during the MF operation and secured the visibility duringfocus confirmation of an object.

A description will be given of the control relating to the AF display insteps S306 to S308 of FIG. 3 . Referring now to FIG. 13 , a descriptionwill be given of the AF frame display menu that can be set in step S301.FIG. 13 explains the AF frame display menu. In this embodiment, the AFframe display menu has three patterns: (A) limited display of AFassisting frames, (B) display of connected AF assisting frame, and (C)full display of AF assisting frames. In a case where the “focusingoperation during AF” is performed, the AF target area is displayed andnotified while the MF operation is stopped, and the MF assisting framefor the MF operation is displayed. Each characteristic will be describedbelow.

(A) Limited Display of AF Assisting Frames

AF assisting frames 1301 are displayed “during AF.” The AF assistingframes 1301 are displayed in the AF target area. In the “focusingoperation during AF,” MF assisting frames 1302 are displayed. The MFassisting frames 1302 are displayed in the AF target area when the MFoperation is stopped. Beneficially, there is no display differencebetween “during AF” and “focusing operation during AF,” and MF and AFare seamlessly connected. However, in a case where the “focusingoperation during AF” is performed, the MF assisting display area is sosmall that the targeted object may not be focused by the MF operation.

(B) Display of Connected AF Assisting Frame

“During AF,” an AF assisting frame 1303, which is made by connecting theAF assisting frames, is displayed in the AF target area. In a case wherethe “focusing operation during AF” is performed, MF assisting frames1305 are displayed on the full screen. An AF frame 1304 is superimposedon the AF targeted area when the MF operation is stopped. Beneficially,in a case where the “focusing operation during AF” is performed, the MFassisting frames are displayed on the entire surface during the MFoperation, and it is highly likely that a targeted object can be focusedby the MF operation. In addition, since the MF display and the AFdisplay have different shapes, the operability of the user can beimproved. However, both the MF and the AF are displayed in the “focusingoperation during AF,” the visibility of the object may be lowered.

(C) Full Display of AF Assisting Frames

“During AF,” AF assisting frames 1306 are displayed on the entirepartial area as the in-focus portion of the screen. In a case where the“focusing operation during AF” is performed, MF assisting frames 1308are displayed on the full screen. An AF target area 1307 is notified tothe user by changing the shape of the MF assisting frame. Beneficially,since the in-focus frames can be displayed even “during AF,” the usercan confirm an in-focused state of an arbitrary object. However, sincein-focus frames are displayed in the entire area “during AF,” it may bedifficult to recognize an object as the AF target. In addition, in acase where the AF target area is notified by changing the shape of theMF assisting frame in the “focusing operation during AF,” the user maynot be able to recognize it depending on the shape. For example, in acase where the MF assisting frame is displayed as a solid-linerectangle, the MF assisting frame in the AF target area can bedistinguished by changing a line width, the number of lines, the solidline into a dotted line, or the size, but this embodiment is not limitedto this example.

Referring now to FIGS. 14A to 14H, a description will be given ofdisplay control in a case where (B) display of a connected AF assistingframe is selected as an example among the three patterns. FIGS. 14A to14H explain the AF assisting frames and the MF assisting frames.

FIG. 14A illustrates the in-focus state on an object A1401 during AF ina scene where the object A1401 and object B1402 exist. FIG. 14Billustrates the start of the MF operation for focusing on the objectB1402 from the in-focus state on the object A1401. FIG. 14C illustratesthe end of the MF operation where the in-focus position has been movedto the object B1402. FIG. 14D illustrates the resume of AF with theobject B1402. FIGS. 14E to 14H illustrate different display variations.FIG. 14A illustrates a connected AF frame 1403 on the object A1401. Whenthe MF operation is started, the screen transitions to FIG. 14B, inwhich MF assisting frames 1405 are displayed, and a connected AF frame1404 indicating the AF target area is displayed if the MF operation isfinished. In a case where the user performs the MF operation relying onthe MF assisting frames 1405 and moves the in-focus position to theobject B1402, the screen transitions to FIG. 14C. During the MFoperation, the MF assisting frames 1405 and 1407 have in-focus display,front blur display, back blur display, or the like, as described insteps S603 to S608. Therefore, the MF operability is improved, and theuser can smoothly focus on the targeted object B1402.

The user confirms that a connected AF frame 1406 indicating the AFtarget area is displayed on object B1402, which is the targeted object,and ends the MF operation. Thereby, the state transitions to FIG. 14D,in which a connected AF frame 1408 is displayed on the object B1402 andAF resumes. Thus, in a case where the “focusing operation during AF” isperformed, the operability of the user improves for both the MF and AFthrough the “display of the AF target area when the MF operation isstopped,” and the “display of the MF assisting frames for the MFoperation.”

As a display pattern different from that described above, in step S301,the camera settings may be made changeable of (C) “display/non-displayof AF frame” “during AF” and (D) “display/non-display of AF subframe” inthe “focusing operation during AF”. (C) is for users who do not needdisplay during AF, and (D) is for users who wish to select an AF targetarea from a plurality of areas.

FIG. 14E illustrates the “non-display of the AF frame” during AF. Theconnected AF frame 1403 is hidden. Thereby, the visibility of the objectduring AF is improved. FIG. 14F illustrates a state at the start of theMF operation. In this state, a connected AF frame 1404 and MF assistingframes 1405 are displayed in and around the AF target area. FIG. 14Gillustrates the “display of AF subframes” at the end of the MFoperation. In a case where there are a plurality of in-focus frames, aprimary AF frame (AF main frame) and a secondary AF frame (AF subframe)are determined according to the continuous areas and positions of thein-focus frames. In this case, the primary AF frame is the connected AFframe 1406 and the secondary AF frame is the AF frame 1409. The user canselect the primary AF target area and secondary AF target area using thecamera operation unit or the like. Thereby, the object selection freedomexpands, and the AF operability can be improved. Similarly to FIG. 14E,FIG. 14G illustrates the “non-display of the AF frame” during AF, inwhich a connected AF frame 1408 is hidden.

Referring now to FIG. 11 , a description will be given of the AFassisting frame setting processing in step S307. FIG. 11 is a flowchartillustrating the AF assisting frame setting processing.

First, in step S1101, the camera control unit 119 acquires the in-focusstate of each phase-difference detecting frame. Next, in step S1102, thecamera control unit 119 turns on the in-focus flag for all framesdetermined to be in focus. Next, in step S1103, the camera control unit119 performs in-focus flag correction processing. This processing isprocessing of correcting a frame with missing an in-focus flag and anisolated in-focus flag in a matrix-shaped focus detection result asillustrated in FIGS. 4A, 4B, and 4C. The in-focus flag correctionprocessing is performed so as to treat, as a connected surface, frameshaving turned on in-focus flags as many as possible.

The order of priority of objects to be the AF target includes conditionssuch as an object having a large area and the center of gravity of thearea being located at the center of the screen. Correcting the in-focusflag can improve the localizing accuracy of the AF target. For example,a method of correcting a frame with a missing in-focus flag includes amethod of monitoring the presence or absence of in-focus flags in fouradjacent frames in the up, down, left, and right directions of a framewhose in-focus flag is turned off (central correction candidate frame).In this method, in a case where the in-focus flags for three or moreframes are turned on, the missing portion can be corrected by changingthe frame (central correction candidate frame) whose in-focus flag isturned off so that the in-focus flag is turned on.

A method of correcting an isolated in-focus flag includes a method ofmonitoring the presence or absence of in-focus flags for four framesadjacent to each other in the up, down, left, and right directions of aframe whose in-focus flag is turned on (isolated candidate frame). In acase where at least two or more frames have in-focus flags that are notturned on, this method can perform processing for correcting isolatedin-focus flags by changing the frame whose in-focus flag is turned on(isolated candidate frame) so that the in-focus flag is turned off. Thiscorrection method is illustrative, and another correction method may beused.

Next, in step S1104, the camera control unit 119 determines whether ornot there is a focusing operation. In a case where there is a focusingoperation, the flow proceeds to step S1105. In a case where there is nofocusing operation, the flow proceeds to step S1108.

Steps S1105 to S1107 are processing in a case where the focusingoperation is performed during AF. In step S1105, the camera control unit119 specifies the AF target area from the in-focus flag. This processingcounts the area of consecutive in-focus flags, sets the area with thelargest area as the primary AF target area, and sets the area with thesecond largest area as the secondary AF target area. In a case wherethey have almost the same areas, the area whose center of gravity isclose to the center of the screen may be set as the primary area. In acase where it is difficult to specify the area of the in-focus flag, thearea may be divided using color information on the image, and the AFtarget area may be specified by the position of the center of gravity inthe area, the area of the in-focus flag, or the like. In addition, thisembodiment may set an area to be evaluated (5×5 frames, etc.), evaluateall in-focus states on the entire screen in that evaluation unit, countthe number of frames whose in-focus flags are turned on, and set thelargest frame close to the center to the AF target area. Even in a casewhere the AF target area cannot be specified by any of the abovemethods, the center of the screen may be set as the AF target area.

Next, in step S1106, the camera control unit 119 performs processing toturn on the AF assisting frame display flag for the specified, primaryAF target area. Next, in step S1107, the camera control unit 119performs processing to turn on the AF assisting subframe display flagfor the specified, secondary AF target area, and ends this flow.

As described above, the camera control unit 119 changes the display ofdisplay items in a case where the user is performing the MF operationduring AF. In a case where the user is performing the MF operationduring AF, the camera control unit 119 may superimpose an AF frameindicating an AF target range for focus control by AF and a displayitem. The camera control unit 119 may display a display item in an AFtarget range where focus control is performed by AF in a case where theuser is performing the MF operation during AF.

Steps S1108 to S1112 are processing during AF. In step S1108, the cameracontrol unit 119 determines whether or not it is just after the end ofthe focusing operation. In a case where the focusing operation has justended, the flow proceeds to step S1109. In a case where it is not justafter the focusing operation ends, the flow proceeds to step S1110. Instep S1109, the camera control unit 119 starts tracking the AF targetarea at the end of the focusing operation as the AF target during AF.

Next, in step S1110, the camera control unit 119 automatically tracksthe AF target area as the object moves. Automatic tracking can berealized using the known technology. For example, this embodiment mayacquire the in-focus state of each phase-difference detecting frame,monitor the vicinity of the AF target area, and repeatedly move the AFtarget area to the detecting frame closest to the in-focus state.Thereby, the AF area can track the object movement on the screen.Alternatively, this embodiment can provide automatic tracking usingcolor information in addition to the in-focus state of thephase-difference detecting frame. Tracking becomes available by storingas a tracking source image an image of the AF target area at the end ofthe focusing operation, by performing when a new image is obtained,pattern matching between the new image and the tracking source image,and by selecting part with the highest correlation as the AF targetarea.

Next, in step S1111, the camera control unit 119 drives the focus lensso that the AF target area is in focus. Next, in step S1112, the cameracontrol unit 119 performs processing to turn on the AF assisting framedisplay flag in the AF target area, and ends the control. Thisprocessing is processing for displaying a frame in the AF area duringAF. The secondary AF frame is not displayed during AF because it issufficient to display the AF frame only for the object for which thein-focus state is to be maintained.

Referring now to FIGS. 12A and 12B, a description will be given of theAF assisting frame display processing in step S308. FIGS. 12A and 12Bare flowcharts illustrating the AF assisting frame display processing.This processing is divided into three types according to the displaytypes described with reference to FIG. 13 : (A) limited display of an AFassisting frame, (B) display of a connected AF assisting frame, and (C)full display of an AF assisting frame. This embodiment will discuss theabove three types, but the effects of this embodiment can be obtainedeven with display methods other than the above display method, as longas two conditions of (1) displaying the AF target area when the MFoperation is stopped and (2) displaying the assisting frames for the MFoperation are satisfied.

First, in step S1201, the camera control unit 119 determines whether ornot the camera setting in step S301 is (A) limited display of the AFassisting frame. In a case where the camera setting is (A) limiteddisplay of the AF assisting frame, the flow proceeds to step S1202. In acase where the camera setting is not (A) limited display of AF assistingframe, the flow proceeds to step S1209. In step S1202, the cameracontrol unit 119 determines whether or not there is a focusingoperation. In a case where there is the focusing operation, the flowproceeds to step S1203. In a case where there is no focusing operation,the flow proceeds to step S1205.

The processing in steps S1203 and S1204 is a mode in which the AFassisting frame is displayed in a case where the focusing operation isperformed during AF. Therefore, the AF target area portion of the MFassisting frame is displayed, and the rest is hidden. In step S1203, thecamera control unit 119 determines whether the AF assisting framedisplay flag is turned on. In a case where the AF assisting framedisplay flag is turned on, the flow proceeds to step S1208. In a casewhere the AF assisting frame display flag is not turned on, the flowproceeds to step S1204. In step S1204, the camera control unit 119 setsthe non-display of the MF assisting frame, and the flow proceeds to stepS1208.

Next, the processing of steps S1205 to S1207 is a mode in which only theAF assisting frame is displayed during AF. Therefore, the MF assistingframes are hidden and only the AF assisting frame in the AF target areais displayed. In step S1205, the camera control unit 119 hides the MFassisting frames. Next, in step S1206, the camera control unit 119determines whether or not the AF assisting frame display flag is turnedon. In a case where the AF assisting frame display flag is turned on,the flow proceeds to step S1207. In a case where the AF assisting framedisplay flag is not turned on, the flow proceeds to step S1208. In stepS1207, the camera control unit 119 performs processing to display the AFassisting frame, and proceeds to step S1208.

In step S1209, the camera control unit 119 determines whether or not thecamera setting in step S301 is (B) display of a connected AF assistingframe. In a case where the camera setting is (B) the display of theconnected AF assisting frame, the flow proceeds to step S1210. In a casewhere the camera setting is not (B) the display of the connected AFassisting frame, the flow proceeds to step S1221. In step S1210, thecamera control unit 119 determines whether or not there is a focusingoperation. In a case where there is the focusing operation, the flowproceeds to step S1211. In a case where there is no focusing operation,the flow proceeds to step S1217.

The processing of steps S1211 to S1216 is a mode for displaying an AFframe connecting the MF assisting frames and the AF assisting frame in acase where the focusing operation is performed during AF. This isprocessing that displays the MF assisting frames as they are, and inaddition, superimposes the AF frame. In step S1211, the camera controlunit 119 determines whether the AF assisting frame display flag isturned on. In a case where the AF assisting frame display flag is turnedon, the flow proceeds to step S1212. In a case where the AF assistingframe display flag is not turned on, the flow proceeds to step S1213. Instep S1212, the camera control unit 119 displays the AF assisting frame,and the flow proceeds to step S1215. In step S1215, the camera controlunit 119 displays the primary AF frame by connecting the AF assistingframes, and hides the AF assisting frame. Next, in step S1216, thecamera control unit 119 connects and displays the AF assisting subframesto display the secondary AF frame, and hides the AF assisting subframes.

The processing in steps S1217 to S1220 is a mode for displaying aconnected AF frame during AF. This is processing to hide the MFassisting frames and display only the connected AF frame. In step S1217,the camera control unit 119 hides the MF assisting frame. Next, in stepS1218, the camera control unit 119 determines whether or not the AFassisting frame display flag is turned on. In a case where the AFassisting frame display flag is turned on, the flow proceeds to stepS1219. In a case where the AF assisting frame display flag is not turnedon, the flow proceeds to step S1208. In step S1219, the camera controlunit 119 displays the AF assisting frame. Next, in step S1220, thecamera control unit 119 displays the connected AF assisting frame in theAF target area and hides the AF assisting frames. The display processingof the connected AF assisting frame will be described below withreference to FIGS. 15A to 15E.

In step S1221, the camera control unit 119 determines whether the camerasetting in step S301 is (C) full display of the AF assisting frame. In acase where the camera setting is (C) full display of the AF assistingframe, the flow proceeds to step S1222. On the other hand, in a casewhere the camera setting is not (C) full display of the AF assistingframe, the flow proceeds to step S1208. In step S1222, the cameracontrol unit 119 determines whether or not there is a focusingoperation. In a case where there is the focusing operation, the flowproceeds to step S1223. In a case where there is no focusing operation,the flow proceeds to step S1225.

The processing in steps S1223 and S1224 is a mode for changing theshapes of the MF assisting frames only for the MF assisting frame andthe AF target area in a case where the focusing operation is performedduring AF. Therefore, the MF assisting frames are displayed as they are,and in addition, the shape of the AF target area is changed. In stepS1223, the camera control unit 119 determines whether or not the AFassisting frame display flag is turned on. In a case where the AFassisting frame is to be displayed, the flow proceeds to step S1224. Onthe other hand, in a case where the AF assisting frame is not to bedisplayed, the flow proceeds to step S1208. In step S1224, the cameracontrol unit 119 changes and displays the shapes of the MF assistingframes, and proceeds to step S1208.

The processing of steps S1225 to S1227 is a mode for displaying framesthat is in focus on the entire screen during AF. Therefore, thisprocessing is processing that hides the MF assisting frames and displaysonly the in-focus AF assisting frame. In step S1225, the camera controlunit 119 hides the MF assisting frames. Next, in step S1226, the cameracontrol unit 119 determines whether the focus display flag set in stepS1102 is turned on. In a case where the in-focus display flag is turnedon, the flow proceeds to step S1227. In a case where the focus displayflag is not turned on, the flow proceeds to step S1208. In step S1227,the camera control unit 119 performs processing for displaying the AFassisting frame, and proceeds to step S1208.

In step S1208, the camera control unit 119 determines whether theprocessing for all the phase-difference detecting frames has ended. In acase where the processing for all the phase-difference detecting frameshas ended, the display processing is finished. On the other hand, in acase where the processing for all the phase-difference detecting frameshas not yet ended, the flow proceeds to step S1201 to repeat the displayprocessing.

Referring now to FIGS. 15A to 15E, a description will be given of (B)display of a connected AF assisting frame (steps S1209 to S1216) in FIG.13 . FIGS. 15A to 15E explain the display of the connected AF assistingframe.

FIG. 15A illustrates AF assisting frames 1501 displayed in a portionwhere the in-focus flag set in step S1102 is turned on. FIG. 15Billustrates AF assisting frames 1502 displayed in a case where the AFassisting frame display flag is turned on only in the AF target area setin step S1106. FIG. 15C is a first example of the connected AF frame,and illustrates a primary AF frame 1504 that is made by connecting theAF assisting frames set in step S1215. This displays the connectedprimary AF frame so as to include all the AF assisting frames. In a casewhere the primary AF frame 1504 is displayed, the visibility improveswhere the AF assisting frame 1502 is hidden, so the AF assisting frames1502 are changed to hidden AF assisting frames 1503.

FIG. 15D is a second example of the connected AF frame, and illustratesa primary AF frame 1505 made by connecting the AF assisting frames setin step S1215. This connected display forms the primary AF frame so asto trace the outer circumference of the AF assisting frames. FIG. 15E isa third example of the connected AF frame, and illustrates a primary AFframe 1506 made by connecting the AF assisting frames set in step S1215.In this connected display, the center of the primary AF frame accordswith the calculated center of gravity of the areas of the AF assistingframes. The AF frame size is variable according to the area.

This embodiment has presented three patterns of the connected display,but various display forms are available depending on the AF assistingframe. The connected display of the subframes in the AF area isdisplayed based on the AF assisting subframes, similarly to the primaryAF frame. As described above, in a case where the “focusing operationduring AF” is performed, this embodiment performs the “display of the AFtarget area when the MF operation is stopped” and the “display of the MFassisting frames for the MF operation,” improving the operability of theuser for both the MF and AF.

Each embodiment can provide a display control apparatus, a displaycontrol method, and a storage medium, each of which can improve theoperability of focusing during MF operation and secure the visibility inconfirming a focus state of an object. Applying focus assisting displayduring the MF operation during AF can provide seamless focus assistingdisplay between AF and MF.

Other Embodiments

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer-executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing thecomputer-executable instructions from the storage medium to perform thefunctions of one or more of the above-described embodiment(s) and/orcontrolling the one or more circuits to perform the functions of one ormore of the above-described embodiment(s). The computer may comprise oneor more processors (e.g., central processing unit (CPU), microprocessing unit (MPU)) and may include a network of separate computersor separate processors to read out and execute the computer-executableinstructions. The computer-executable instructions may be provided tothe computer, for example, from a network or the storage medium. Thestorage medium may include, for example, one or more of a hard disk, arandom-access memory (RAM), a read-only memory (ROM), a storage ofdistributed computing systems, an optical disc (such as a compact disc(CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flashmemory device, a memory card, and the like.

While the disclosure has been described with reference to embodiments,it is to be understood that the disclosure is not limited to thedisclosed embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2022-109427, filed on Jul. 7, 2022, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A control apparatus configured to control adisplay unit configured to superimpose and display a captured imagecaptured by an image sensor and display items indicating in-focusdegrees of a plurality of focus detecting areas in the captured image,the control apparatus comprising: a memory that stores a set ofinstructions; and a processor configured to execute the set ofinstructions to change a display state of the display items according toa state of a manual focus operation by a user.
 2. The control apparatusaccording to claim 1, wherein the processor is configured to: determinewhether or not a user is performing a manual focus operation; and changethe display state of the display items according to a determinationresult.
 3. The control apparatus according to claim 1, wherein thedisplay state includes a display ratio of the display items per unitarea, and wherein the processor is configured to: set the display ratioto a first ratio in a case where the user is not performing the manualfocus operation, and set the display ratio to a second ratio higher thanthe first ratio in a case where the user is performing the manual focusoperation.
 4. The control apparatus according to claim 1, wherein thedisplay state includes the display number of display items, and whereinthe processor is configured to: set the display number to a firstdisplay number in a case where the user is not performing the manualfocus operation, and set the display number to a second display numberlarger than the first display number in a case where the user isperforming the manual focus operation.
 5. The control apparatusaccording to claim 1, wherein the processor is configured to: displaythe display items indicating that the in-focus degrees are in-focusstates on the display unit, and thin out and displays on the displayunit, part of the display items in a case where the user is notperforming the manual focus operation.
 6. The control apparatus,according to claim 1, wherein the processor is configured to: displaythe display items indicating that the in-focus degrees are in-focusstates on the display unit, and connect the plurality of display itemsand display a connected display item on the display unit in a case wherethe user is not performing the manual focus operation.
 7. The controlapparatus according to claim 1, wherein the display state includes adisplay gap between the display items, and wherein the processor isconfigured to: set the display gap to a first gap in a case where theuser is not performing the manual focus operation, and set the displaygap to a second gap narrower than the first gap in a case where the useris performing the manual focus operation.
 8. The control apparatusaccording to claim 1, wherein the display state includes a range of thein-focus degrees of the display items, and wherein the processor isconfigured to: set the range to a first range in a case where the useris not performing the manual focus operation, and set the range to asecond range wider than the first range in a case where the user isperforming the manual focus operation.
 9. The control apparatusaccording to claim 8, wherein the processor is configured to determinewhether or not to change the range according to the display number ofdisplay items.
 10. The control apparatus according to claim 8, whereinthe display state includes colors of the display items, and wherein theprocessor is configured to: set the colors to a first color in a casewhere the user is not performing the manual focus operation, and set thecolors to a second color different from the first color in a case wherethe user is performing the manual focus operation.
 11. The controlapparatus according to claim 8, wherein the display state includesshapes of the display items, and wherein the processor is configured to:set the shapes to a first shape in a case where the user is notperforming the manual focus operation, and set the shapes to a secondshape different from the first shape in a case where the user isperforming the manual focus operation.
 12. The control apparatusaccording to claim 1, wherein the processor is configured to change adisplay range of the display items according to a zoom operation by theuser.
 13. The control apparatus according to claim 1, wherein theprocessor is configured to change a display range of the display itemsaccording to a panning or tilting operation by the user.
 14. The controlapparatus according to claim 1, wherein the processor is configured tochange display of the display items in a case where the user isperforming the manual focus operation during autofocus.
 15. The controlapparatus according to claim 14, wherein the processor is configured todisplay and superimpose an AF frame indicating an AF target range forfocus control by the autofocus and the display items in a case where theuser is performing the manual focus operation during the autofocus. 16.The control apparatus according to claim 14, wherein the processor isconfigured to display the display items in an AF target range for focuscontrol by the autofocus in a case where the user is performing themanual focus operation during the autofocus.
 17. The control apparatusaccording to claim 1, wherein the display state includes an offsetamount for a focus detection result, and wherein the processor isconfigured to: acquire the offset amount according to a focus controlamount by the manual focus operation, and change display of the displayitems by subtracting the offset amount from the focus detection result.18. The control apparatus according to claim 1, wherein the displaystate includes an offset amount for a focus detection result, andwherein the processor is configured to: acquire the offset amountaccording to a display delay amount of the display items on the displayunit, and change display of the display items by subtracting the offsetamount from the focus detection result.
 19. The control apparatusaccording to claim 1, wherein the display state includes an offsetamount for a focus detection result, and wherein the processor isconfigured to: acquire the offset amount set by the user, and changedisplay of the display items by subtracting the offset amount from thefocus detection result.
 20. A control method configured to control adisplay unit configured to superimpose and display a captured imagecaptured by an image sensor and display items indicating in-focusdegrees of a plurality of focus detecting areas in the captured image,the control method comprising: changing a display state of the displayitems according to a state of a manual focus operation by a user.
 21. Anon-transitory computer-readable storage medium storing a program thatcauses a computer to execute the display control method according toclaim
 20. 22. An image pickup apparatus comprising: a display unitconfigured to superimpose and display a captured image captured by animage sensor and display items indicating in-focus degrees of aplurality of focus detecting areas in the captured image; and a controlapparatus configured to control the display unit, wherein the controlapparatus includes: a memory that stores a set of instructions, and atleast one processor that executes the set of instructions to change adisplay state of the display items according to a state of a manualfocus operation by a user.